The latest test failures are from test_XarrayZarrRecipe.py::test_process, and stem from the fact that in #349 we punted on how to hash preprocessor functions. In this PR, we're calling recipe.get_execution_context() (which includes a recipe.sha256() call) on every execution, so the tests which include preprocessors fail. The options would seem to be:

1. Find a way to deterministically hash Python functions. (I've previously wondered if the function objects __code__ attribute might offer some path forward here...)
2. Don't set a "pangeo-forge:recipe_hash" if the instance contains a preprocessor.
3. Do set a "pangeo-forge:recipe_hash" for recipes with a preprocessor, but exclude the preprocessor from the calculation. We might think of this as thematically similar to the unsafe_hash option for built-in dataclasses.

Preprocessors are a very useful feature, as (will be) hashes, so my preference would be to try to find a way to achieve option 1 if possible. I will take a brief pause on this PR to look into this, but if this seems infeasible, as a backup plan we could move forward with option 2 (not setting hashes for recipes with preprocessors). It seems to me that option 3 could lead to some very confusing situations which we'd prefer to avoid.

Find a way to deterministically hash Python functions. (I've previously wondered if the function objects code attribute might offer some path forward here...)

Let's use inspect for this.

import inspect

def foo(a, b):
    a += 1
    return a + b

inspect.getsource(foo)

I think that is a fine solution for now. It's a little bit brittle because it is sensitive to changes in formatting of the function. However, it should be sufficient to detect when a defined function has not changed at all.

I was not able to get __code__.co_code to work, as in this example

def foo(a, b):
    a += 1
    return a + b

def bar(a, b):
    a += 2
    return a + b

# this assertion fails
assert foo.__code__.co_code != bar.__code__.co_code

Thanks for the review, Ryan. inspect.getsource is very cool, hadn't seen it before! I think that's the last thing we need here, will add that now.